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EXECUTIVE SUMMARY 


RMC Research used a case study approach to document implementation of Imagine Language & 
Literacy and a correlational design to examine the relationship between student program usage and 
student academic achievement outcomes for students in Grades K-2. The study was conducted during 
the 2018/19 academic year at eight schools in two school districts in the Southeastern United States. 
The case study approach involved interviews with administrators, focus groups with teachers, and 
observations of Imagine Language & Literacy in six schools to gather detailed information about 
implementation. For all study schools, RMC Research and Imagine Learning collected usage data and 
administrative data including demographic information and scores on the formative Dynamic Indicators 
of Basic Early Literacy Skills (DIBELS) Next assessment for all students in Grades K-2 to examine the 
relationship between student program usage and student academic achievement. Findings focus on 
implementation factors and the impact of the amount of instruction received in Imagine Language & 
Literacy on spring DIBELS Next scores. 


FINDINGS 


In general, teachers and administrators found the professional development provided to be sufficient 
for beginning implementation of Imagine Language & Literacy. Teachers and administrators reported 
that the initial professional development they received from Imagine Learning was informative and 
sufficient to get them started using Imagine Language & Literacy. 


Teachers suggested additional training or documentation on using Imagine Language & Literacy. 
Teachers indicated they would have liked additional training on how to use many of the components of 
Imagine Language & Literacy and that they would specifically like training regarding the student-level 
reports available in the system. Additionally, some teachers requested a “Frequently Asked Questions 
(FAQ)” or other documentation be available to aid them in finding information within Imagine Language 
& Literacy such as reports and additional resources. 


Implementation was hindered by technology problems. All teachers and administrators reported 
experiencing technology problems, many of which negatively affected implementation of Imagine 
Language & Literacy. 


Students enjoyed engaging with Imagine Language & Literacy. Teachers and administrators reported 
their students enjoyed working in Imagine Language & Literacy and the program was engaging for 
students. 


Fewer than 60% of students in each grade level met the recommendation for average number of 
minutes per week of participation. Imagine Learning recommends that students in Kindergarten engage 
with Imagine Language & Literacy for at least 45 minutes per week and students in Grades 1 and 2 
engage with the program for at least 60 minutes per week. On average, 56% of Kindergarten students, 
47% of students in Grade 1, and 24% of students in Grade 2 met the recommended total minutes per 
week. 
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Average student participation varied by school. Across the schools in the study, participation in Imagine 
Language & Literacy varied. School averages for total weeks ranged from 15 to 30 weeks, averages for 
total minutes ranged from 647 to just under 2,000 minutes, and averages for activity minutes ranged 
from 507 to 1,604 minutes. 


Extent of participation (dosage) was related to performance on the spring DIBELS Next assessment for 
students in Kindergarten. All indicators of dosage were positively related to spring scores on the DIBELS 
Next assessment for students in Kindergarten. For example, after accounting for prior achievement and 
demographic characteristics, an increase in participation of 167 total minutes, 125 activity minutes, 2.7 
weeks, 3.5 minutes per week, or 2.8 activity minutes per week was associated with a one-point increase 
on the DIBELS Next Letter Naming Fluency test for students in Kindergarten. For students in the analytic 
sample, a one-point increase on this test would be related to an additional 1.8% of students meeting 
DIBELS Next benchmark scores for progress and 2.6% fewer students being identified as at risk. 


Extent of participation (dosage) was related to performance on the spring DIBELS Next assessment for 
students Grade 1. All indicators of dosage were positively related to spring scores on the DIBELS Next 
assessment for students in Grade 1. For students in Grade 1, an increase of 125 total minutes, 111 
activity minutes, 1.4 weeks, 3.7 minutes per week, or 3.2 activity minutes per week was associated with 
a one-point increase in DIBELS Next Nonsense Word Fluency: Correct Letter Sounds score and increases 
in participation of 250 total minutes, 250 activity minutes, 3.2 weeks, 7.4 minutes per week, or 6.2 
activity minutes per week were associated with a similar increase on the DIBELS Next Nonsense Word 
Fluency: Whole Words Correct score. For students in the analytic sample, a one-point increase on this 
test would be related to an additional half percent of students meeting DIBELS Next benchmark scores 
for progress and 0.2% fewer students being identified as at risk on the DIBELS Next Nonsense Word 
Fluency: Correct Letter Sounds test and an additional 1.1% of students meeting DIBELS Next benchmark 
scores for progress and 2.8% fewer students being identified as at risk on the DIBELS Next Nonsense 
Word Fluency: Whole Word Correct test. 


Extent of participation (dosage) was not significantly related to performance on the spring DIBELS 
Next assessment for students in Grade 2. No indicators of dosage were found to be significantly related 
to spring scores on the DIBELS Next assessment for students in Grade 2. Overall implementation was 


lowest in Grade 2, with only 24% of students meeting optimal dosage recommendations. This finding 
suggests that weak implementation reduces the likelihood of detecting student outcomes. 


RECOMMENDATIONS 


Based on findings from this study, RMC Research offers the following recommendations to guide 
program development and improvement, and assess the impact of Imagine Language & Literacy. 


1. Provide additional hands-on experience with Imagine Language & Literacy programming and 
reports during the Getting Started training. 


2. Document technology issues and share common solutions or a manual for common issues. 
3. Continue monitoring implementation throughout the academic year. 


4. Continue encouraging sites to implement programming and meet usage recommendations. 
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INTRODUCTION 


This section provides background information about the Imagine Language & Literacy program and the 
2018/19 study of program implementation and outcomes. 


BACKGROUND 


Imagine Language & Literacy is a computer-based instructional program that helps students acquire 
English language and literacy skills through individualized instruction. The intervention is targeted 
towards elementary grades, emergent readers, struggling readers, English language learners, and 
students with disabilities. Imagine Learning contracted with RMC Research to conduct a study to 
describe implementation of Imagine Language & Literacy in a case study approach and to document 
correlational evidence of the relationship between program usage and student academic achievement 
for struggling readers using the program in Grades K-2. 


IMAGINE LANGUAGE & LITERACY PROGRAM 


Imagine Language & Literacy is designed as a comprehensive language and literacy intervention for 
elementary students. It works to close achievement gaps at an accelerated rate by automatically 
creating a unique computer-based curriculum for each student based on his or her needs. The program 
incorporates over 4,300 activities founded on research about English language learning, game-based 
learning, and acquisition of literacy skills to help students develop language and literacy proficiency. 
Language and literacy skills covered by the program include grammar, phonemic awareness, phonics, 
academic vocabulary, fluency, and reading comprehension. Imagine Language & Literacy provides 
strategic first-language support for English language learners and offers support materials in 15 
languages. Imagine Language & Literacy assesses student performance and learning needs through 
benchmark and progress monitoring assessment tools. The program is designed to be interactive 
through multiple modalities to support student engagement. Educators receive regular reports through 
ongoing assessments and Lexile® measures. 


Imagine Language & Literacy includes the following key instructional components: 


Thorough language and literacy instruction. Students receive systematic and explicit instruction in the 
five key areas of literacy as outlined by the National Reading Panel: phonics, phonemic awareness, 
fluency, vocabulary, and comprehension. In addition to instruction in these five areas, Imagine 
Language & Literacy also provides English language instruction, building skills in listening and speaking. 
Students also gain test-taking skills through regular use of the program. More than 4,300 engaging 
activities provide students with opportunities to learn and practice skills. 


e Phonemic awareness: Students learn to recognize phonemes and how to blend phonemes to 
create words. Students also learn how to segment phonemes and learn the concept of rhyme. 


1 Lexile measure is the numeric representation of an individual’s reading ability or a text’s readability (or difficulty). 
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English language learners receive explicit instruction regarding English phonemes that do not 
occur in their first language. 


e Phonics: Students learn important phonics skills, including letter recognition, letter sounds, and 
word recognition. Students practice these skills in a selection of engaging activities. 


e Fluency: Students practice fluency by reading a variety of books and creating audio recordings of 
their reading. The books range in difficulty from simple decodable texts to more complex natural 
texts. Content areas include science, social studies, literary genres (such as myths, plays, poems, 
and tall tales), biography, narrative, and mathematics. Selections are paired, so that one article 
provides background knowledge for the other. Additionally, each story is written at two levels of 
difficulty: easy and advanced. Recordings are saved, allowing teachers to track student progress. 


e Vocabulary: Students learn basic and academic vocabulary through explicit instruction and 
through context. New vocabulary enables students to better understand science, mathematics, 
and social studies as well as communication arts. Translations of key vocabulary words are 
available for English language learners through first-language support. 


e Reading comprehension: Students learn comprehension strategies and practice answering 
comprehension questions about main idea, cause and effect, problem and solution, author’s 
purpose, and more. 


e Listening and speaking: Students learn conversational phrases and academic vocabulary. 
Students also practice listening comprehension skills by listening to texts that are read aloud 
and then answering a comprehension question. 


e Test-taking skills: The comprehension questions students encounter in the program are similar 
to questions encountered on state tests. Students learn important skills that assist in test taking, 
such as how to answer intuitive questions and how to answer literal questions. 


First-language support. Imagine Language & Literacy provides strategic first-language support for 
English language learners, including translations of vocabulary words, customized activities, 
explanations of difficult concepts, and more. This support gradually fades as the student progresses. 
First-language support is available in Arabic, Cantonese, English, French, Haitian Creole, Hmong, 
Japanese, Korean, Mandarin, Marshallese, Portuguese, Russian, Spanish, Somali, Tagalog, and 
Vietnamese. Print materials are also available in the following languages: Burmese, Farsi, Karen, Polish, 
and Urdu. 


Differentiated instruction for each student. The program is automatically differentiated for each 
student. It recognizes if students need remediation or if they can be placed on an accelerated path. 
Curriculum is automatically adjusted to ensure students are provided with the content they need to 
support measurable growth. This removes much of the pressure from teachers to differentiate 
instruction for their diverse classes. 


High level of engagement. The program uses multiple modalities and employs high-quality art, video, 
and music, as well as chants, games, and personalizable characters to keep students involved. These 


RMC Research Corporation, Denver, CO 2 Study of Imagine Language & Literacy 
in Grades K-2 


engagement strategies help to ensure that students remember what they learned, so they can use their 
newly acquired skills in all content areas and on exams. 


PRIOR RESEARCH 


Several prior studies provide evidence of the effects of Imagine Language & Literacy. For example, 
Imagine Learning is one of four vendors that currently participates in the Utah Board of Education Early 
Intervention Reading Software Program. Annual evaluations of this program examine the effect of the 
overall program on student achievement, implementation of each vendor’s program, and impact of each 
vendor’s program. Recent findings suggest about half of students who participated in Imagine Language 
& Literacy met minimum participation recommendations and that the program has had a significantly 
positive effect on student achievement with effect sizes” ranging from 0.04 to 0.24 for students in 
Grades K-2, based on comparison with nonparticipating students (Hobbs, 2016; Hobbs, 2017; Hobbs, 
2018). 


A Regression Discontinuity Design study conducted in the 2013/14 academic year demonstrated that 
elementary students who engaged with Imagine Language & Literacy for at least 20 hours 
demonstrated greater growth on the Scantron Reading Foundations assessment than students in a 
comparison condition (Heller & Carter, 2015). The authors note the sample size was a limiting factor in 
interpreting results, suggesting the study was underpowered. A correlational study conducted during 
the 2016/17 academic year identified a positive relationship between and minutes of program usage per 
week and Lexile growth (Imagine Learning, 2018-a). 


A series of case studies using academic achievement outcomes in single user districts or schools 
documented correlational relationships between Imagine Language & Literacy program usage and 
school and student-level outcomes in California (Imagine Learning, 2018-b), Connecticut (Imagine 
Learning, 2018-c), Florida (Imagine Learning, 2018-d), an Oklahoma school district (Imagine Learning, 
2018-e), and two Texas school districts (Imagine Learning, 2018-f; Imagine Learning, 2018-g). A study 
examining the proportion of students in Grades 3-5 who met English language arts proficiency levels on 
the Florida State Assessment during the 2016/17 year found that schools implementing Imagine 
Language & Literacy had higher proportions of proficient students (Imagine Learning, 2018-d). 


RMC Research conducted a quasi-experimental design study with propensity score matching to evaluate 
the impact of Imagine Language & Literacy on the reading achievement of low-performing students in 
Grades 1-5 (Espel, Meyer, & Weston-Sementelli, 2019; Espel & Meyer, 2019). The study was conducted 
during the 2017/18 academic year at 63 schools in five school districts representing three states in the 
Midwest, Northeast, and Southeast regions of the United States. Imagine Language & Literacy was 
implemented in intervention schools during the spring 2018 semester. Findings focus on the impact of 
the program on student reading achievement, based on formative and summative student reading 
achievement measures. Analyses also examined variation in program implementation and the 
relationship between measures of program implementation and reading achievement on formative and 
summative assessments. The study demonstrated that implementation varied. No statistically significant 
effect of participation in /magine Language & Literacy on reading achievement was found. The study 
found a mix of null and negative relationships between dosage and student outcome scores. 


2 Effect size (ES) is a name given to a family of indices that measure the magnitude of a treatment effect, represented by 
differences in outcomes across groups. Unlike significance tests, these indices are independent of sample size. An effect size 
(Cohen’s d) of .8 or greater is considered a large effect. 
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Another recent quasi-experimental design study which used propensity score matching compared 
growth in reading scores on the State of Texas Assessments of Academic Readiness (STAAR) for fourth- 
and fifth-grade students who used Imagine Language & Literacy and those who did not (SEG 
Measurement, 2018). This study was conducted during the 2017/18 academic year at three districts in 
Texas and included 2,346 students. After controlling for initial ability, the study found that students who 
used Imagine Language & Literacy scored higher than those who did not engage with the program. The 
study found the effect size for fourth-grade students to be .24 and the effect size for fifth-grade 
students to be .14. 


These studies provide initial and promising evidence of the impact of Imagine Language & Literacy. 
Imagine Learning wishes to develop additional and more rigorous evidence of the relationship between 
participation in Imagine Language & Literacy and student achievement outcomes. 


THIS STUDY 


The Every Student Succeeds Act (ESSA), which is the reauthorization of the Elementary and Secondary 
Education Act describing federal regulations for education in the United States, directs state and local 
educational agencies, school administrators, and educators to provide evidence-based interventions to 
support student learning. This mixed-methods study used a case study approach and correlational 
analyses examining the relationship between program implementation and outcomes. The study was 
designed with the potential to provide promising evidence (Tier 3) of effectiveness under ESSA? and 
provides descriptive information about Imagine Language & Literacy implementation that may provide 
feedback for program improvement and inform priorities for future research. 


As of late 2019, “promising evidence” is defined in U.S. Department of Education regulations (§77.1: 
Definitions that apply to all Department programs)‘ as follows: 


Promising evidence means that there is evidence of the effectiveness of a key project 
component in improving a relevant outcome, based on a relevant finding from one of the 
following: (i) A practice guide prepared by WWC reporting a “strong evidence base” or 
“moderate evidence base” for the corresponding practice guide recommendation, (ii) An 
intervention report prepared by the WWC reporting a “positive effect” or “potentially 
positive effect” on a relevant outcome with no reporting of a “negative effect” or 
“potentially negative effect” on a relevant outcome; or (iii) A single study assessed by the 
Department, as appropriate, that—(A) Is an experimental study, a quasi-experimental 
design study, or a well-designed and well-implemented correlational study with 
statistical controls for selection bias (e.g., a study using regression methods to account 
for differences between a treatment group and a comparison group); and (B) Includes at 
least one statistically significant and positive (i.e., favorable) effect on a relevant 
outcome. 


The two national organizations that review evidence in terms of the ESSA evidence levels, however, 
define promising evidence somewhat differently. Evidence for ESSA, based at the Center for Research 
and Reform in Education at Johns Hopkins, identifies mathematics and reading programs as evidence- 


3 ESSA defines promising evidence as being supported by “at least one well-designed and well-implemented correlational study 
with statistical controls for selection bias (Elementary and Secondary Education Act of 1966, §8101 (21)(A), 2018. 
4 Available here: https://www.ecfr.gov/cgi-bin/text-idx?node=34:1.1.1.1.24&rgn=div5 
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based according to ESSA evidence levels based on reviews only of experimental and quasi-experimental 
group design studies (Evidence for ESSA, n.d.). Studies that do not meet the “strong” or “moderate” 
evidence levels are eligible to be used as evidence for “promising” interventions. The What Works 
Clearinghouse only reviews studies that meet the “strong” or “moderate” evidence levels and offers no 
“promising” evidence designation. 


The present study builds on prior research and was designed to document program implementation and 
the relationship between implementation and outcomes, using a sample of students from two districts 
in two Southeast region states during the fall and spring semesters of the 2018/19 school year. The 
study examines the relationship between extent of participation (dosage) and student achievement in 
Grades K-2 on formative assessments. Student achievement was measured using data from fall and 
spring Dynamic Indicators of Basic Early Literacy (DIBELS) Next Benchmark Assessments referred to in 
this report as “DIBELS Next”. 


The next section presents study methodology, followed by a presentation of findings. Findings first 
present descriptive information related to the case studies of each site and then present correlational 
analyses documenting the relationship between program implementation and student outcomes. The 
final section presents conclusions and recommendations. 
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METHODOLOGY 


This section describes the study research question, design, intervention, quantitative and qualitative 
data, and analyses. 


RESEARCH QUESTION 


This study was designed to answer the question: How does Imagine Language & Literacy 
implementation vary, and how does variation in implementation relate to formative student 
achievement outcomes?? 


DESIGN 


RMC Research used a case study approach to document implementation of Imagine Language & 
Literacy and a correlational design to examine the relationship between student program dosage® and 
student academic achievement outcomes for students of all reading levels in Grades K-2. The study was 
designed so that in all study schools, all students in Grades K-2 participated in the intervention during 
the entire 2018/19 academic year. The case study approach involved site visits, interviews with 
administrators, focus groups with teachers, and observations of Imaging Language & Literacy in six 
schools to gather detailed information about implementation. For all study schools, RMC Research 
obtained Imagine Language & Literacy dosage data and administrative data including demographic 
information and scores on the formative DIBELS Next assessment for all students in Grades K-2 to 
examine the relationship between student program dosage and academic achievement outcomes. 


IMAGINE LANGUAGE & LITERACY INTERVENTION 


Participating districts and schools committed to implementation of Imagine Language & Literacy using 
recommended dosage; provision of technology support, equipment/devices, and personnel for program 
implementation; communication and support to school personnel; implementation monitoring; and 
provision of administrative data including student demographics and assessment scores. Free licenses 
for the Imagine Language & Literacy program were provided to participating schools. 


Students in Grades K-2 in study schools participated in a whole-school intervention. For this study, 
Imagine Learning recommended that on average, Kindergarten students use the program for 45 minutes 
per week and students in Grade 1 and Grade 2 use the program for 60 minutes per week. Weekly 
dosage was monitored through the online Imagine Language & Literacy platform. 


5 RMC Research initially designed the study in collaboration with Imagine Learning to examine the impact of Imagine Language 
& Literacy on K-2 student outcomes with additional research questions. Challenges with implementation beyond the control of 
the study team (e.g., natural disasters, school closings) and challenges with recruitment impeded the completion of the impact 
study. The current study was adapted from the original design and provides pilot data to inform future rigorous research 
studies. 

6 The term dosage refers to the extent of student participation in the program, and includes time spent in the program, activity 
time spent in the program, weeks spent implementing the program, and minutes per week spent in the program. 
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Teachers received key supports including: 


e access to Imagine University with online training materials; 

e training provided by Customer Success Managers; 

e access to Imagine Learning’s teacher care call center; 

e onsite visits by Customer Success Managers; 

e weekly student usage reports and Action Area Tools to support lesson planning; 
e guidance related to planning and implementation, and 

e technical support. 


Training for teachers and administrators provided information about setting up classes, using the 
program, and using program resources to monitor student performance. Teachers were instructed to 
allot time during regular instruction for students to participate in the Imagine Language & Literacy 
program using computers. Students first completed a Language and Literacy Benchmark test that placed 
them in the lesson sequence that most closely aligned with their needs. Imagine Learning’s formative 
assessments were administered throughout the intervention. 


SAMPLE 


Imagine Learning recruited two public school districts for the study from the Southeast region of the 
United States. Schools in each district were randomly selected to participate in the study and provided 
data for all students who participated in Imagine Language & Literacy. Both districts represented urban 
(small city) settings. Data from eight schools were included in quantitative analyses and data from six 
schools were included in qualitative analyses (both schools from one district and four randomly-selected 
schools of six schools in the other district). Students were excluded from the analytic sample for 
quantitative analyses if they used Imagine Language & Literacy but were not in one of the identified 
study schools, or if they were not in Grades K-2. 


QUALITATIVE DATA 


Site visits were conducted at six schools in two districts. Data were collected from December 11-12, 
2018 at two schools in one district and from February 4-6, 2019 at four schools in the second district. 


Qualitative data Site visits included: 


e Classroom observations. In each school, a sample of classrooms was observed implementing 
Imagine Language & Literacy. A total of 22 classes were observed across the six schools. 
Classroom observations were scripted during visits. 


e Teacher focus groups. At each school, focus groups were conducted with Imagine Language & 
Literacy teachers to assess experiences with implementation, including engagement in program 
activities such as technical support for the intervention technology, interaction with Imagine 
Learning support teams, training, usage review, and others. Some teachers were also site 
coordinators who had primary responsibility for implementation of Imagine Language & 
Literacy. Teachers were also asked to characterize perceived strengths and challenges, 
perceptions of the extent to which Imagine Language & Literacy met identified goals, other 
language and literacy supports students received, and perceived impacts on students involved 
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with Imagine Language & Literacy. Seven teacher focus groups were conducted, each of which 
included two to six teachers. A lab monitor and English as Second Language (ESL) teacher 
implementing Imagine Language & Literacy with their K-2 students were also interviewed using 
the same protocol. 


e School administrator interviews. The principal at each of the six schools participating in the case 
study site visit was interviewed. These interviews collected information about reasons for 
participation in the study, roles and responsibilities of the site coordinators, the type and quality 
of professional development provided to teachers, the nature of Imagine Language & Literacy 
implementation in their school, the support and resources received from Imagine Learning and 
their district, factors facilitating and impeding implementation, alignment with curriculum, their 
impressions of Imagine Language & Literacy, and expected outcomes from participation in the 
program. 


QUANTITATIVE DATA 
Data from eight schools were included in quantitative analyses and included the following: 


DIBELS Next. The DIBELS Next Benchmark Assessment, developed by the University of Oregon Center on 
Teaching and Learning, was used to collect data for all students in fall 2018 and spring 2019. DIBELS Next 
is a standardized, evidence-based, criterion-referenced assessment of basic early literacy skills. It 
measures overall literacy and reading skills with a composite score. Subscales used for this study 
include: 


e =Letter Naming Fluency 

e Nonsense Word Fluency: Correct Letter Sounds 
e Nonsense Word Fluency: Whole Words Correct 
e Oral Reading Fluency: Words Correct 

e Oral Reading Fluency: Accuracy 


Subscales were available for various grade levels across both study districts. Letter Naming Fluency was 
collected for Kindergarten, Nonsense Word Fluency was collected for Grade 1, and Oral Reading Fluency 
was collected for Grade 2. Each subscale yields a raw score and a percentile rank; an indicator of 
students’ progress towards meeting DIBELS Next benchmark goals related to students’ expected literacy 
and reading levels for an overall composite score and subscales; and an indicator of whether or not the 
student would be identified as being at risk for reading difficulty. 


DIBELS Next is designed to be administered up to three times per year for universal screening and/or to 
monitor the effectiveness of school instructional supports. The assessment may be administered in a 
paper-and-pencil format or through online tablet applications, with an examiner. The assessment takes 
5 to 10 minutes to complete. Prior studies have demonstrated the reliability and validity of the DIBELS 
Next Benchmark Assessment. DIBELS Next demonstrates alternate-form reliability with coefficients 
above .90 in Grades 1-6, and .66 in Kindergarten. Composite scores have high inter-rater reliability (.94 
to .99). Predictive validity ranges from .48-.80. Concurrent validity is .73-.80 in Grades 1-5, and .40 in 
Kindergarten (Dewey, Kaminski, & Good, 2014). Cutpoints for DIBELS Next benchmarks and identifying 
students who are at risk for reading challenges and in need for support were drawn from University of 
Oregon Center on Teaching and Learning (2012). 
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Student, School, and District Demographics. RMC Research and Imagine Learning collected student, 
school, and district data expected to be related to outcome measures and of priority interest to Imagine 
Learning. These data elements included: school; district; gender; grade; race/ethnicity; free or reduced- 
price lunch eligibility; homeless status; migrant status; English language learner status; and special 
education status. 


Imagine Language & Literacy dosage. Imagine Learning provided RMC Research with student-level 
information about participation in Imagine Language & Literacy. Data elements used to examine the 
relationship between dosage and achievement outcomes included: the total minutes students were 
engaged with Imagine Language & Literacy each week (used to calculate the number of weeks of 
program implementation and average minutes per week) and the total minutes students engaged in 
educational activities within Imagine Language & Literacy. Student total time and activity time were 
provided by week, and student data were excluded for a given week if the total minutes of activity time 
in that week was less than 5 minutes. Fewer than nine students each week had scores that were outliers 
(more than three standard deviations above the mean);’ for these students, minutes of activity time 
each week and total weeks of implementation were adjusted to three standard deviations above the 
mean. The total time students spent in /magine Language & Literacy and the total activity time were 
calculated using these adjusted values. Students were credited for having participated in a week of 
Imagine Language & Literacy if they had logged at least 5 minutes?® in the system in that week. Students 
who engaged with Imagine Language & Literacy for less than one calendar month? (n = 28)*° were 
excluded from the analytic sample. 


The analytic sample for quantitative analyses consisted of 1,375 students including 450 students in 
Kindergarten (K), 467 students in Grade 1, and 458 students in Grade 2 for whom fall and spring DIBELS 
Next scores were available, and for whom student dosage and demographic data were available. 
Exhibits 1 through 3 show that most students were White or African American. Across all grade levels 
the percentage of White students was roughly equal with between 57% and 60% per grade level. The 
percentage of African American students was also consistent across grade levels with between 33% and 
36% of students identifying as African American. Between 45% and 58% of students who participated in 
Imagine Language & Literacy were female. Exhibit 4 provides additional demographic information by 
grade level. Over half of students in each grade level were eligible for free or reduced-price lunch with 
between 56% and 67% eligible. The percentage of students identified for special education services 
ranged from 8% (Kindergarten) to just over 11% (Grade 1). The percentage of students identified as 
English language learners ranged from just under 10% in Grade 2 to 12% of students in Grade 1. Across 
all grade levels, less than 2% of students were identified as homeless or as migrant students. 


7 Standard deviation (SD) is a measure of how spread out a set of values is. Higher standard deviations indicate greater 
variability in data across respondents. 

Mean or average value is a measure of central tendency computed by adding a set of values and dividing the sum by the total 
number of values. 

8 In collaboration with Imagine Learning, five-minutes was selected as a minimum usage cut-off threshold because fewer 
minutes was determined to not represent meaningful usage. Less than five-minutes of usage would likely not give students 
enough time to complete Imagine Language & Literacy activities. 

° Calculated as at least 1 month elapsing between a student’s first and last Imagine Language & Literacy session. This cut off 
was selected in collaboration with Imagine Learning to ensure that students in the analytic sample engaged with Imagine 
Language & Literacy for a meaningful amount of time; many students were in schools with high student mobility. 

10 is the total number in a sample. n is the number in a subsample. 

1 The initial sample in study schools consisted of 1,756 students, the analytic sample represents 78.3% of this initial sample. 
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EXHIBIT 1. PERCENT OF STUDENTS BY RACE/ETHNICITY AND GENDER IN KINDERGARTEN 


Race/Ethnicity 


White 


African American 


Hispanic/Latino 


Asian American 


Other 


60.0% 


Kindergarten Students 


Gender 


EXHIBIT 2. PERCENT OF STUDENTS BY RACE/ETHNICITY AND GENDER IN GRADE 1 


Race/Ethnicity 


White 


African American 


Hispanic/Latino 


Asian American 


Other 


RMC Research Corporation, Denver, CO 


57.2% 


Female 


44.8% 


First Grade Students 


10 


Gender 


f 467 


Study of Imagine Language & Literacy 
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EXHIBIT 3. PERCENT OF STUDENTS BY RACE/ETHNICITY AND GENDER IN GRADE 2 
Race/Ethnicity Gender 

White 60.3% 

African American 


Hispanic/Latino 


Asian American 


Other 
P 458 
Second Grade Students 
EXHIBIT 4. ADDITIONAL STUDENT CHARACTERISTICS BY GRADE LEVEL 
Student-Level Characteristics Kindergarten Grade 1 Grade 2 
Number (N) 450 467 458 
English Language Learner (%) 10.7 12.0 9.6 
Eligible for Free or Reduced-Price Lunch (%) 67.2 62.3 55.7 
Homeless Students (%) 1.1 0.4 0.2 
Migrant Students (%) 0.8 1.5 0.0 
Students Identified for Special Education Services (%) 8.2 11.3 10.3 
ANALYSES 


Focus group and interview data were coded and analyzed using an approach that closely follows 
methods explicated by Miles, Huberman, and Saldafia (2014). This approach emphasizes well-defined 
study variables so that the common themes can be identified. Qualitative data were analyzed to identify 
trends in responses to each question. 


Descriptive analyses were conducted to identify the nature and extent of variation in implementation 
across study sites. These findings were reported using ranges, frequencies, percentages, means, and 
standard deviations. Linear regression was used to determine the relationship of dosage to student 
outcomes using DIBELS Next scores. The approach for this correlational analysis involved building a 
model that explained variance in student achievement scores with fall DIBELS Next scores and student- 
level covariates. Measures of dosage were tested for their contribution to explaining student DIBELS 
Next literacy scores. 
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Linear regression analyses** were conducted using SPSS Version 24.0 to assess the relationship of 
variations in dosage to student outcomes measured using DIBELS Next scores. Regression analyses were 
conducted for each grade level and each dosage indicator independently. Dosage was measured using 
the total number of minutes, total number of activity minutes, average minutes per week, average 
activity minutes per week, the total number of weeks students engaged with Imagine Language & 
Literacy, and meeting the optimal dosage threshold. Several student-level variables, including pretest 
achievement (fall 2018 DIBELS Next scores) and student demographic data elements (special education 
status; race/ethnicity [White or nonWhite]; English language learner status) were used as covariates in 
the correlational analyses to identify the relationship between program dosage and student outcomes 
above and beyond the influence of other factors. For these analyses, achievement scores are presented 
in their original raw score format for DIBELS Next. All other variables are dichotomous or “dummy” 
variables that have values of either zero or one.?? 


12 Regression analysis is a statistical procedure that examines the relationship among two or more variables. This type of 
analysis allows for examination of the relative contribution of multiple independent variables to outcomes of interest. 

13 For example, students who were not identified as English language learners have a value of zero on this variable; those who 
were identified as English language learners have a value of one on this variable. 
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FINDINGS 


This section summarizes findings of analyses to address the research question. Findings from the case 
study are presented first, followed by descriptive analyses of implementation and correlational analyses 
relating program dosage to student outcomes. 


How DOES IMPLEMENTATION OF IMAGINE LANGUAGE & LITERACY VARY? 


CLASSROOM OBSERVATIONS 


Across the two districts and six schools, three different implementation models for Imagine Language & 
Literacy were observed. In these models, students engaged with Imagine Language & Literacy: 


e In whole classrooms with a 1:1 student-to-device ratio and the classroom teacher present; 

e Inacomputer lab with a 1:1 student-to-computer ratio and either a lab monitor or classroom 
teacher present. In some cases, this model was implemented with small groups of students 
because the computer lab or classroom could not accommodate the entire class at once; and 

e In stations in classrooms with fewer devices than students and the classroom teacher present. 


In all cases, students were told when it was time to work with Imagine Language & Literacy and did so 
independently at their own device. Devices observed being used during the site visits included desktop 
computers, laptops, chrome books, and iPads. Almost all students stayed in the computer lab or 
classroom for the duration of their Imagine Language & Literacy time; however, a few students were 
pulled for small group instruction or testing during this time. During implementation, teachers engaged 
in various activities, including grading student work, pulling students for small group instruction or 
testing, and monitoring for technology issues. Teachers were not observed helping students complete 
the instructional programming. 


The amount of time students needed to log in and begin engaging in Imagine Language & Literacy 
activities varied. For example, in some cases, if a student did not remember their username and/or 
password they needed to wait for a teacher or lab monitor to help them. In other cases, students came 
into the lab and referred to a card with their username and password which allowed them to log in 
quickly. 


Most students attended to the program and actively engaged with the tasks presented. When the 
program prompted students to record themselves, some students barely whispered into the 
microphone and had to be prompted to speak louder while others shouted into the microphones. A few 
students were observed speaking gibberish or saying “blah blah blah” repeatedly when prompted to 
record, but most spoke along with the program and were heard sounding out words that were difficult 
for them. Some teachers and lab monitors prompted students to listen to their recordings prior to 
submitting them to ensure they could be understood. Students were audibly excited when they 
advanced in the program. As they advanced in the program, they earned coins and were able to spend 
them on games and personalizing their character. In most cases, the teacher or lab monitor roamed the 
room offering technical help when needed and encouragement to students. A few students did not 
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engage with the program. For these students, a “timeout” screen appeared after a few minutes, 
generally prompting the teacher or lab monitor to attempt to get the student to engage with the 
program. 


A number of technology issues were observed including: students being unable to log in, students being 
logged out in the middle of a task, and problems with microphones, headphones, and Internet access. In 
many cases, these problems resulted in students being unable to log onto Imagine Language & Literacy 
or being unable to record themselves. In one lab session, either the Internet or Imagine Language & 
Literacy experienced a problem and a whole class was unable to log onto the program. The lab monitor 
attempted to get students into the program and called Imagine Learning technical support; however, 
the lab monitor was unable to resolve the issue. Students seemed most frustrated when they were 
suddenly logged out of the program in the middle of an activity. Some students logged right back in, 
with many losing the progress they made during that session. Other students complained to their 
teacher or lab monitor and were either sent back to class or told to work on another activity until their 
time ended. Another common issue was that session times were shorter than the class period resulting 
in sessions ending prior to the end of the scheduled Imagine Language & Literacy time. In these 
instances, some students logged back in and others began another activity or returned to class. 


Almost all teachers had their students use Imagine Language & Literacy during a scheduled period each 
day. Only one teacher was not using the program at the designated time during the site visits. Several 
classes or labs had substitutes during observations; students tended to have challenges logging in to the 
program when these individuals were present. 


TEACHER FOCUS GROUPS AND SCHOOL ADMINISTRATOR INTERVIEWS 


Teachers and school administrators responded to questions about a range of topics including 
professional development, implementation, support, monitoring, student engagement, student results, 
and recommendations for improvement. Teachers involved in the focus groups had varied experience 
ranging from a first-year computer lab monitor to a teacher with more than 20 years of experience. In 
addition, a number of teachers used Imagine Language & Literacy in the past. Administrators 
interviewed during the case studies had varying amounts of experience ranging from a first-year 
administrator to an administrator with more than 20 years of experience. 


The majority of the administrators agreed for their schools to participate in the study because they had 
previous positive experience with Imagine Language & Literacy. Those who lacked prior experience with 
Imagine Language & Literacy generally agreed to participate out of a desire to improve student learning. 
All of the administrators interviewed served as the main point of contact for Imagine Learning at their 
schools. In this role, administrators answered teacher questions, handled technical issues, and 
troubleshot other issues as they arose. For example, one administrator entered all teacher and student 
information into the program, assigned usernames and passwords, and personally outfitted lab and 
computers with microphones for students to use. 


Professional Development. Teachers reported the timing of the training varied with teachers receiving 
training either in the summer, at the beginning of the school year, or after the academic year began. 
Training sessions occurred online or through video conferencing. Most teachers indicated that the 
training was very informative and provided sufficient information to start implementation. However, a 
few teachers felt the trainings were rushed and did not spend enough time on how to interpret reports. 
Some teachers also reported that the trainings did not afford them the opportunity to explore the 
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system, that no manual was provided, and information about the program was often relayed by 
teachers who used it in previous years. A number of teachers indicated they had learned much of what 
they knew about working with Imagine Language & Literacy during past experiences with the program. 
All teachers expressed a desire for more training with a focus on how to pull and interpret student data 
reports. One teacher suggested the training could be improved through the use of small group 
instruction. Another recommendation was that a manual or “Frequently Asked Questions (FAQ” 
document be provided to help teachers navigate the platform and access additional resources. Below 
are some representative quotes from teachers about the training: 


The training was a basic review and introduction to [Imagine Language & Literacy]. It 
wasn’t really a training on how to use it. We could have used training on how to pull 
data, look data up by class, and move students between classes. — Teacher 


The training was rushed, but it helped us to get started. — Teacher 


The training was good for what was available, but we really needed to be able to log in 
during the training to be able to [experience the system] and look for data. — Teacher 


We figured [the program out] as we went along, it would have been useful to [learn 
how] to listen to students’ recordings. — Teacher 


We could have used training on how and when [students] redo lessons and explanations 
of the parts of the tests and assessments and the dashboard. We can see student 
growth, but we can’t explain it, | don’t know how to pull that information. — Teacher 


In general, administrators experienced the same professional development presented to teachers that 
provided a general overview how to get started. Administrators also learned how to set up the program 
by entering student and teacher information into the program. Administrator ratings of the overall 
quality of training varied across sites. Most administrators agreed the training provided sufficient 
information to support implementation. They stated that the training staff was professional and 
provided exceptional customer service when needed. Administrators also had a few recommendations 
for improvement. They felt the training could have been improved if more time was provided to review 
program components and included more hands-on activities. All the of administrators felt that an 
additional training component after the intervention started would have been beneficial to provide 
more hands-on information and technical support. Below are some representative quotes from 
administrators about the training: 


The training was good, but we had not set up accounts yet so we couldn’t work with it. 
We had to follow up with more information on how to access [Imagine Language & 
Literacy]. — Administrator 


[The training] was great, [there is] never enough time. . . It was definitely adequate to 
get us started and [Imagine Learning staff] are always available [to help us]. 
— Administrator 


The training was good, | wish they had included the amount of time each grade level was 
supposed to [use the program]. This was shared around Christmas. — Administrator 
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A second training a month [into implementation] would have been great to go over data 
and available resources. — Administrator 


Monitoring. Teachers monitored student program implementation and growth by viewing the student 
dashboards through Imagine Language & Literacy and looking at the weekly reports sent by Imagine 
Learning documenting student program usage. Teachers indicated they liked that Imagine Language & 
Literacy instruction was differentiated to meet the needs of individual students. For some, monitoring 
student progress was difficult due to the varied levels of materials students were engaging within the 
system, while others indicated that monitoring progress was easy because program activities aligned 
with the curriculum. A few teachers indicated that when students worked on material in Imagine 
Language & Literacy that was ahead of topics covered in their classroom, they referred to this 
experience when the topic was introduced. 


Another group of teachers indicated that the reports highlighted the need to increase grammar 
instruction in their grade level. Some teachers reported difficulty monitoring growth because of 
inaccurate information on Imagine Language & Literacy reports. For example, a few teachers reported 
that Imagine Language & Literacy reports sometimes indicated that a student had not started a lesson 
when the lesson was already completed. Others, however, indicated that the action areas noted for 
students on the reports were accurate. 


Support for Implementation. Teachers listed several ways that implementation of Imagine Language & 
Literacy was supported by Imagine Learning and their districts. For example, when headphones with 
embedded microphones were not available for students to use, Imagine Learning provided them to 
schools. In addition, technical support was provided by Imagine Learning. Teachers and lab monitors 
reported difficulties helping students while they were using Imagine Language & Literacy due to the 
amount of time needed and the necessity to leave a student logged in while experiencing a problem. 
Some teachers reported they received support from their district when asked, while others reported 
receiving limited support. 


Administrators varied in their assessments of the quality of support received from Imagine Learning. 
Many administrators appreciated the quick responses from Imagine Learning staff while others reported 
challenges and limited communication with Imagine Learning about implementing the program in their 
school. Some administrators noted they received district support by receiving Chromebooks, 
headphones, and iPads for students to use. Administrators indicated that having the support of their 
teachers, and teachers who were open and willing to try Imagine Learning facilitated effective 
implementation of the program. 


Implementation Challenges. Teachers reported that all students used Imagine Language & Literacy in 
the classroom or in a computer lab. Technological issues were reported by all teachers who participated 
in the focus groups. Teachers and administrators reported that the following issues affected overall 
implementation of the program the ability for students to create recordings: 


e Students lacking headphones; 

e Headphones being incompatible with devices; 

e Computers freezing; 

e Students being logged off in the middle of a session; and 
e Microphones breaking. 
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Teachers and lab monitors reported that because of how and when Imagine Language & Literacy saved 
student progress, many students had to re-complete tasks. This challenge was reported as a frequent 
issue that occurred when students had to log out before a lesson was completed or when they were 
automatically logged out of the program. For example, students frequently had to close the program 
and move to their next scheduled activity prior to finishing the timed lesson. To decrease the need to 
repeat information, students were either sent back to class early, or they logged into a different 
(mathematics) program activity for the remainder of their /magine Language & Literacy time. 


The only non-technology related implementation difficulty mentioned by teachers was that Imagine 
Language & Literacy interfered with the time needed for other instructional programs in their school. 
For example, one district had recently purchased a phonics program that teachers wanted to use. One 
school was concurrently implementing three different online programs including one targeting 
computer literacy and one for mathematics. Below are some representative quotes from teachers 
regarding implementation challenges. 


The [technology requirements] for headphones and microphones are a disadvantage. 
We needed to spend our money on headphones for the classroom. — Teacher 


Our biggest issues are students being logged off, the number of computers available, 
and when the sound won’t work, that’s a big issue because they [students] have to be 
able to hear what the program is saying to them. — Teacher 


[Imagine Language & Literacy] does not save a student’s progress mid-lesson, this is a 
problem when they get logged off or [don’t get to finish] within a class. — Teacher 


Some administrators reported that their school had difficulties meeting the recommended minute 
requirements for students, with many students only engaging with Imagine Language & Literacy for 30 
minutes per week. One reason identified for this difficulty was the lack of available computer lab time or 
availability of computer carts. At the time of the site visits, administrators were working with Imagine 
Learning to find ways for students to meet the recommended dosage. Below are some representative 
quotes from administrators regarding implementation challenges. 


Headphones have been a big challenge for us. Not all are compatible with each 
computer. The recording trips us up. — Administrator 


Technology factors, scheduling issues, time allocations, the number of available 
computers and the technical requirements have [been difficult for us]. — Administrator 


Our greatest challenge is other district requirements, we can’t have all of the computer lab time 
we’d like, we are required to administer computerized benchmark assessments and [another 
digital program]. — Administrator 


Student Outcomes. Teachers reported students very much enjoyed using Imagine Language & Literacy 
and they asked to use the program when using other computer-based learning software. Teachers also 
reported that Imagine Language & Literacy was easy for students to use and was very engaging for 
students. One teacher highlighted that the read-aloud feature helped give students more confidence. An 
overwhelming majority of teachers said they believed that participation in Imagine Language & Literacy 
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positively affected their students. Some said that students who participated in the program 
demonstrated higher vocabulary and reading scores, as well as increased computer fluency. 


Teachers reported they expected to see student growth on computer-based formative assessments like 
DIBELS Next because Imagine Language & Literacy covered relevant topics and provided students with 
an opportunity to interact with computer-based learning systems. Overall, administrators believed that 
Imagine Language & Literacy aligned with their school curriculum. One administrator noted that 
Imagine Learning could be improved if there was a way to align program components with current 
lesson plans for the teacher. 


Teachers suggested that benefits may be accelerated for English language learners and more advanced 
students. The ability of Imagine Language & Literacy to provide scaffolded instruction in a student’s 
native language was seen as a benefit among teachers who were unable to provide this type of 
instruction. Teachers reported rapid growth in achievement for these students. In addition, teachers 
reported that Imagine Language & Literacy helped to prepare English language learner students for 
state tests. They explained that state tests require students to record themselves speaking and Imagine 
Language & Literacy provided practice with this skill. Teachers also reported benefits for advanced 
learners, for whom providing challenging material was sometime difficult while simultaneously meeting 
the needs of struggling students. Below are some representative quotes from teachers regarding 
student outcomes: 


We hope to see growth in all students, and a lot of them showed gains on the interim 
assessment. — Teacher 


Imagine Language & Literacy is very interactive and engaging, as long as students are 
not logged out. The reading aloud component really gives them a sense of confidence. 
The students don’t even realize they are learning. — Teacher 


The increased computer use has helped students to get used to using them. This will be a 
benefit for [other digital assessments]. — Teacher 


Imagine Learning provides students with good exposure to reading. They do it every day. 
— Teacher 


Imagine Language & Literacy meets students where they are... and the oral reading is 
very helpful for students. — Teacher 


[Imagine Language & Literacy] helps [students] to establish a love of learning ... singing 
rhymes. They tell me all about what they are learning. It definitely keeps them engaged. 
— Teacher 


We have seen some benefits. Some of our low performing students and English language 
learners [have] really benefited from [the program]. — Teacher 


All teachers said they would recommend Imagine Learning to another teacher because it is motivating 
to students, easy for students to use, and does not represent a large additional burden for teachers, for 
example one teacher stated: 
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My favorite thing is that [Imagine Language & Literacy] in not a lot of teacher work... it 
sets a pathway for them. We know it is a quality program and we trust it. — Teacher 


Nearly all school administrators also reported that students very much enjoyed interacting with the 
program. Based on their observations they believed that participating in Imagine Language & Literacy 
resulted in increased phonics skills and improved student confidence in their ability to use new 
vocabulary outside of class. They reported similar benefits for English language learners. Administrators 
said they expected to see some form of growth in each student, whether it is based on state assessment 
scores or students reaching grade-level proficiency. They stated that, compared to other programs, 
Imagine Language & Literacy was easy to use and provided support for a variety of student levels. Below 
are some representative quotes from administrators regarding student outcomes: 


Students really enjoy Imagine Language & Literacy compared to the program we used 
last year. They ask to go and work on it. — Administrator 


Students love it and are excited about learning. We are developing life-long learning. 
— Administrator 


| have seen increased confidence in students as their skills develop. Some are catching on 
quicker to blending and segmenting and picking up new sight words beyond the class 
materials. | think the growth is due to the additional exposure [through Imagine 
Language & Literacy]. — Administrator 


We expect to see growth in our students and an increase in student growth percentiles 
and the number meeting proficiency. — Administrator 


Imagine Language & Literacy has made a positive impact. It helps students at all levels. 
High performing students get to [engage with] new concepts... and ELL [English 
Language Learner] students have their needs met as well. We’ve seen carryover of skills 
into the ELL classrooms. — Administrator 


Imagine Language & Literacy helps [students] with digital literacy and strongly supports 
our instruction. We’ve already seen some very good growth in reading in Grade 1 on 
their interim assessments. — Administrator 


DOSAGE 


Imagine Language & Literacy dosage was measured using total time (minutes) in the program, total 
activity time (minutes), number of weeks of program implementation, average total minutes per week, 
average activity time per week, and meeting the optimal dosage threshold. 


The average total minutes spent in the Imagine Language & Literacy program varied by grade level and 
across schools (Exhibits 5 and 6). Across grade levels, school average total minutes in /magine Language 
& Literacy ranged from 647 to just under 2,000 minutes. Overall, across schools and grade levels, 
students engaged with the program for between 49 and 3,182 minutes (M = 1,268, SD = 624). On 
average, students in Kindergarten and Grade 2 spent just under 1,200 total minutes on the program 
(Kindergarten M = 1,192, SD = 549; Grade 2 M = 1,174, SD = 657). Students in Grade 1 on average used 
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the program for over 1,400 minutes during the school year (Grade 1 M = 1,433, SD = 628). See Exhibit 6 
for a detailed description. 


EXHIBIT 5. AVERAGE TOTAL TIME (MINUTES) STUDENTS SPENT IN 
IMAGINE LANGUAGE & LITERACY BY GRADE LEVEL 


All Grades Kindergarten Grade 1 Grade 2 


Note. The lines represent the standard deviation (SD). 
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EXHIBIT 6. TOTAL TIME (MINUTES) STUDENTS SPENT IN 
IMAGINE LANGUAGE & LITERACY BY SCHOOL AND GRADE LEVEL 


School 
1 2 3 4 5 6 7 8 Total 
Across All Grade Levels 
N 78 156 185 214 154 183 176 229 1375 
Min 289.0 49.4 514.0 127.4 467.0 129.8 518.5 436.0 49.4 
Max 2780.2 2554.3 2795.4 3182.4 2631.0 815.7. 3162.5 2645.3 3182.4 
Mean 985.3 692.7 1497.0 1224.4 1532.2 647.1 1999.6 1367.4 1267.9 
SD 539.1 582.7 399.0 660.1 443.7 85.5 357.4 414.2 624.4 
Kindergarten 
N 31 41 63 82 54 60 50 69 450 
Min 289.0 49.4 514.0 127.4 947.4 516.8 518.5 1025.5 49.4 
Max 1619.5 936.5 2795.4 1611.6 1818.6 815.7. 1998.8 2137.7 2795.4 
Mean 1036.7 341.2 1670.7 856.1 1391.8 676.3 1674.2 1672.1 1192.0 
SD 352.4 205.6 378.9 391.9 197.7 75.0 211.0 248.4 549.0 
Grade 1 
N 29 64 62 56 58 63 63 72 467 
Min 296.5 195.3 1181.1 776.7 467.0 157.9 757.3 685.2 157.9 
Max 1023.0 2554.3 2207.4 3182.4 2631.0 767.0 2402.0 2645.3 3182.4 
Mean 605.9 1253.3 1686.8 1980.0 1519.2 651.4 2004.6 1395.3 1432.8 
SD 182.3 484.9 257.5 592.4 558.0 90.5 267.2 526.0 627.8 
Grade 2 
N 18 51 60 76 42 60 63 88 458 
Min 873.5 56.8 614.6 248.0 676.7 129.8 1502.6 436.0 56.8 
Max 2780.2 955.4 1588.2 2186.3 2495.9 696.5 3162.5 1530.4 3162.5 
Mean 1508.1 271.9 1118.6 1065.1 1730.6 613.6 2252.7 1105.6 1174.2 
SD 708.9 195.2 247.9 462.7 429.7 79.3 321.7. 178.7 657.3 


Average activity minutes spent in the Imagine Language & Literacy program also varied by grade level 
and school (Exhibits 7 and 8). Across grade levels, school average activity minutes in Imagine Language 
& Literacy ranged from 507 to just over 1,600 minutes. Overall, across schools and grade levels, students 
engaged with educational activities in the program for between 37 and 2,432 minutes 

(M = 973, SD = 485). On average, students in Kindergarten and Grade 2 spent approximately 900 activity 
minutes on the program (Kindergarten M = 900, SD = 417; Grade 2 M = 918, SD = 514). On average, 
students in Grade 1 used the program for almost 1,100 activity minutes during the school year (Grade 1 
M = 1,098, SD = 492). See Exhibit 8 for a detailed description. 
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EXHIBIT 7. AVERAGE ACTIVITY TIME (MINUTES) STUDENTS SPENT IN 
IMAGINE LANGUAGE & LITERACY BY GRADE LEVEL 
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EXHIBIT 8. ACTIVITY TIME (MINUTES) STUDENTS SPENT IN 
IMAGINE LANGUAGE & LITERACY BY SCHOOL AND GRADE LEVEL 


School 
1 2 3 4 5 6 7 8 Total 
Across All Grade Levels 
N 78 156 185 214 154 183 176 229 1375 
Min 211.2 36.8 420.9 61.1 274.4 90.4 401.8 330.2 36.8 
Max 2081.0 1819.4 1835.5 2432.4 1964.2 693.3 2405.5 1962.8 2432.4 
Mean 731.7 507.1 1102.7 926.4 1165.5 528.6 1604.2 1052.1 973.0 
SD 401.6 414.1 286.2 503.4 367.9 72.9 290.2 317.9 484.8 
Kindergarten 
N 31 41 63 82 54 60 50 69 450 
Min 211.2 36.8 420.9 61.1 274.4 382.4 401.8 778.0 36.8 
Max 1305.5 768.2 1835.5 1250.9 1403.8 693.3 1637.8 1684.9 1835.5 
Mean 768.8 262.3 1172.7 637.5 1031.5 545.9 1334.6 1290.1 899.9 
SD 283.3 161.2 267.6 299.9 196.4 66.1 186.5 209.7 417.1 
Grade 1 
N 29 64 62 56 58 63 63 72 467 
Min 226.5 125.3 918.9 562.9 334.1 90.4 609.9 412.6 90.4 
Max 822.1 1819.4 1747.7 2432.4 1964.2 649.6 2001.5 1962.8 2432.4 
Mean 474.1 898.3 1276.6 1487.0 1161.0 523.3 1653.8 1033.2 1097.5 
SD 143.4 350.6 203.1 449.9 438.2 79.4 228.2 400.0 492.4 
Grade 2 
N 18 51 60 76 42 60 63 88 458 
Min 574.9 41.4 421.2 213.8 560.0 116.9 1175.1 330.2 41.4 
Max 2081.0 673.2 1234.5 1696.3 1950.1 610.1 2405.5 1178.4 2405.5 
Mean 1082.6 212.9 849.5 825.0 1344.0 516.8 1768.6 881.0 917.9 
SD 555.7 150.1 187.9 371.0 363.5 70.1 263.8 158.4 514.1 
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The average weeks spent in the Imagine Language & Literacy program varied slightly across grade levels 
and schools (Exhibits 9 and 10). Across grade levels, the school average for number of weeks of 
participation in Imagine Language & Literacy ranged from 15 to 30 weeks. Overall, across schools and 
grade levels, students engaged with the program for between 3 and 34 weeks (MM = 24.1, SD = 6.6). 
Average weeks did not vary much by grade level, with Grade 2 students receiving instruction for 

23.0 weeks, Kindergarten students receiving instruction for 24.3 weeks, and Grade 1 students receiving 
instruction for 24.9 weeks, on average. See Exhibit 10 for a detailed description. 


EXHIBIT 9. AVERAGE NUMBER OF WEEKS STUDENTS RECEIVED INSTRUCTION IN 
IMAGINE LANGUAGE & LITERACY BY GRADE LEVEL 
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EXHIBIT 10. WEEKS STUDENTS RECEIVED INSTRUCTION IN 
IMAGINE LANGUAGE & LITERACY BY SCHOOL AND GRADE LEVEL 


School 
1 2 3 4 5 6 7 8 Total 
Across All Grade Levels 
N 78 156 185 214 154 183 176 229 1375 
Min 7.0 3.0 16.0 7.0 10.0 5.0 10.0 18.0 3.0 
Max 32.0 30.0 30.0 31.0 30.0 21.0 32.0 34.0 34.0 
Mean 23.1 14.6 26.3 23.6 25.2 18.7 30.3 28.3 24.1 
SD 5.7 8.5 2.2 5.0 4.7 1.8 2.8 3.3 6.6 
Kindergarten 
N 31 41 63 82 54 60 50 69 450 
Min 7.0 4.0 16.0 7.0 21.0 14.0 10.0 25.0 4.0 
Max 32.0 23.0 30.0 30.0 28.0 21.0 32.0 34.0 34.0 
Mean 24.6 10.6 26.7 22.0 26.2 19.2 30.1 31.9 24.3 
SD 6.3 4.5 2.1 4.9 1.3 1.4 3.0 1.6 6.8 
Grade 1 
N 29 64 62 56 58 63 63 72 467 
Min 9.0 7.0 24.0 17.0 13.0 6.0 11.0 20.0 6.0 
Max 25.0 30.0 29.0 31.0 30.0 20.0 32.0 32.0 32.0 
Mean 19.1 22.7 26.8 28.2 23.6 18.5 30.0 27.1 24.9 
SD 3.5 5.9 1.0 2.9 5.9 1.8 3.7 2.8 5.4 
Grade 2 
N 18 51 60 76 42 60 63 88 458 
Min 21.0 3.0 19.0 9.0 10.0 5.0 26.0 18.0 3.0 
Max 31.0 22.0 30.0 29.0 30.0 20.0 32.0 30.0 32.0 
Mean 26.9 7.5 25.4 22.1 26.2 18.4 30.8 26.6 23.0 
SD 3.3 3.7 2.9 4.3 5.0 2.1 1.3 2.4 7.3 


Average total minutes per week spent in the Imagine Language & Literacy program also varied by grade 
level and school (Exhibits 11 and 12). Across grade levels, school average total minutes per week in 
Imagine Language & Literacy ranged from 41 to just under 66 minutes. Across schools and grade levels, 
average total minutes per week spent in Imagine Language & Literacy ranged from 12 to 110 minutes 
per week (M = 49.9, SD = 16.2). Students in Kindergarten and Grade 2 spent on average between 46 and 
48 minutes per week on the program (Kindergarten M = 46.4, SD = 13.3; Grade 2 

M = 47.9, SD = 16.8). Students in Grade 1 used the program, on average, for the most minutes per week 
(Grade 1 M =55.4, SD = 16.6). See Exhibit 12 for a detailed description. 
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EXHIBIT 11. AVERAGE TOTAL MINUTES PER WEEK STUDENTS SPENT IN 
IMAGINE LANGUAGE & LITERACY BY GRADE LEVEL 
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EXHIBIT 12. TOTAL NUMBER OF MMIINUTES PER WEEK STUDENTS RECEIVED INSTRUCTION IN 
IMAGINE LANGUAGE & LITERACY BY SCHOOL AND GRADE LEVEL 


School 
1 2 3 4 5 6 7 8 Total 
Across All Grade Levels 
N 78 156 185 214 154 183 176 229 1375 
Min 22.8 12.4 32.1 18.2 29.2 26.0 47.5 24.2 12.4 
Max 89.7 85.1 96.4 109.7 99.0 41.2 99.3 87.7 109.7 
Mean 40.6 41.5 56.4 49.0 60.3 34.6 65.9 47.6 49.9 
SD 13.5 16.2 12.5 19.0 12.0 3.1 9.6 10.8 16.2 
Kindergarten 
N 31 41 63 82 54 60 50 69 450 
Min 27.6 12.4 32.1 18.2 36.4 27.2 47.5 38.4 12.4 
Max 52.2 49.7 96.4 59.7 67.4 41.1 64.5 66.5 96.4 
Mean 41.6 30.7 62.1 37.2 53.1 35.2 55.5 52.3 46.4 
SD 6.0 9.8 11.4 11.3 7.2 3.2 3.7 6.5 13.3 
Grade 1 
N 29 64 62 56 58 63 63 72 467 
Min 22.8 27.9 48.8 30.3 29.2 26.3 59.2 29.8 22.8 
Max 44.5 85.1 81.8 109.7 94.0 41.2 75.1 87.7 109.7 
Mean 31.2 54.2 62.8 69.4 62.6 35.2 66.7 50.6 55.4 
SD 4.8 13.0 8.6 17.6 12.1 3.2 3.2 15.1 16.6 
Grade 2 
N 18 51 60 76 42 60 63 88 458 
Min 34.9 16.7 32.4 22.6 45.6 26.0 50.1 24.2 16.7 
Max 89.7 62.6 58.1 80.2 99.0 39.4 99.3 52.8 99.3 
Mean 54.3 34.3 43.7 46.5 66.5 33.3 73.2 41.5 47.9 
SD 19.2 12.8 6.0 14.0 12.3 2.4 9.9 50.1 16.8 


in Grades K-2 


Average activity minutes per week spent in the Imagine Language & Literacy program varied by grade 
level and school (Exhibits 13 and 14). As expected, the number of activity minutes per week was lower 
than the average total time per week. Across grade levels, school average activity minutes per week in 
Imagine Language & Literacy ranged from 28 to just under 53 minutes per week. Across schools and 
grade levels, average activity minutes per week spent in Imagine Language & Literacy ranged from 

9 to 84 minutes per week (MM = 38.4, SD = 12.6). Students in Kindergarten spent about 35 activity 
minutes per week (M = 35.1, SD = 10.1); Grade 2 students spent about 38 activity minutes (M =37.5, 
SD = 13.1), and Grade 1 students spent about 42 activity minutes per week (M =42.4, SD = 13.1). See 
Exhibit 14 for a detailed description. 


EXHIBIT 13. AVERAGE ACTIVITY MINUTES PER WEEK STUDENTS SPENT IN 
IMAGINE LANGUAGE & LITERACY BY GRADE LEVEL 
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EXHIBIT 14. NUMBER OF ACTIVITY MINUTES PER WEEK STUDENTS RECEIVED INSTRUCTION IN 
IMAGINE LANGUAGE & LITERACY BY SCHOOL AND GRADE LEVEL 


School 
1 2 3 4 5 6 7 8 Total 
Across All Grade Levels 
N 78 156 185 214 154 183 176 229 1375 
Min 18.0 9.2 22.2 8.7 10.6 15.1 33.7 18.3 8.7 
Max 67.1 60.7 67.6 83.9 76.0 35.1 77.6 67.4 83.9 
Mean 30.2 31.0 41.6 37.0 45.9 28.2 52.8 36.6 38.4 
SD 9.9 11.7 9.2 14.7 10.8 2.9 7.7 8.2 12.6 
Kindergarten 
N 31 41 63 82 54 60 50 69 450 
Min 21.0 9.2 26.3 8.7 10.6 20.1 33.7 27.1 8.7 
Max 42.1 41.6 67.6 46.3 52.0 35.1 52.8 54.4 67.6 
Mean 30.7 23.6 43.7 27.7 39.4 28.4 44.2 40.4 35.1 
SD 5.3 8.2 8.6 8.9 7.4 3.0 3.8 5.7 10.1 
Grade 1 
N 29 64 62 56 58 63 63 72 467 
Min 18.0 17.9 36.3 22.5 20.9 15.1 47.5 20.0 15.1 
Max 35.8 60.7 64.7 83.9 70.2 34.5 62.6 67.4 83.9 
Mean 24.4 38.9 47.6 52.2 47.7 28.2 55.0 37.4 42.4 
SD 3.9 9.7 7.1 13.6 10.0 3.2 3.1 11.5 13.1 
Grade 2 
N 18 51 60 76 42 60 63 88 458 
Min 24.9 13.6 22.2 18.0 32.3 22.2 40.0 18.3 13.6 
Max 67.1 56.5 44.4 64.0 76.0 34.3 77.6 42.1 77.6 
Mean 38.8 27.0 33.2 35.9 51.8 28.1 57.4 33.0 37.5 
SD 15.4 10.4 4.8 11.2 11.3 2.4 7.8 4.4 13.1 


OPTIMAL DOSAGE 


Imagine Learning recommends that students in Kindergarten engage with Imagine Language & Literacy 
for at least 45 minutes per week and that students in Grade 1 and Grade 2 engage with the program for 
60 minutes per week. Across schools, more than half (56%) of Kindergarten students participated for the 
recommended 45 total minutes per week. Adherence to optimal dosage was lower in Grades 1 and 2 
with less than half (47%) of students in Grade 1 and less than a fourth (24%) of students in Grade 2 
meeting the recommended dosage of 60 minutes per week (Exhibit 15). Average minutes of 
participation also varied substantially across schools. Exhibits 16 through 18 present graphs with the 
data in Exhibit 15 compared to the recommended optimal dosage. 
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EXHIBIT 15. STUDENTS MEETING OPTIMAL DOSAGE BY SCHOOL AND GRADE LEVEL 


Percent of Students Meeting Optimal Dosage 


School 
1 2 3 4 5 6 7 8 Total 
Kindergarten 29.0 9.8 93.7 29.3 83.3 0.0 100.0 85.5 55.6 
Grade 1 0.0 34.4 62.9 69.6 69.0 0.0 98.4 25.0 47.1 
Grade 2 33.3 2.0 0.0 18.4 66.7 0.0 98.4 0.0 23.6 


EXHIBIT 16. MINUTES PER WEEK STUDENTS RECEIVED INSTRUCTION IN 
IMAGINE LANGUAGE & LITERACY BY SCHOOL, KINDERGARTEN 
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EXHIBIT 17. MINUTES PER WEEK STUDENTS RECEIVED INSTRUCTION IN 
IMAGINE LANGUAGE & LITERACY BY SCHOOL, GRADE 1 
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EXHIBIT 18. MINUTES PER WEEK STUDENTS RECEIVED INSTRUCTION IN 
IMAGINE LANGUAGE & LITERACY BY SCHOOL, GRADE 2 


Average Minutes per Week 


School 


=== Recommended Dosage 


As expected, the average number of activity minutes per week was lower than the average total time 
per week. On average, about 77% of total time spent in /magine Language & Literacy was logged as 
activity time, suggesting that, in this case, the optimal activity time™ for students in Kindergarten may 
be approximately 35 minutes (77% of 45 minutes) per week and 46 minutes (77% of 60 minutes) per 
week for students in Grade 1 and Grade 2. Over half of students in Kindergarten (53%), just over 44% of 
Grade 1 students, and approximately 24% of Grade 2 students met these average participation times per 
week. 


How DOES VARIATION IN IMPLEMENTATION RELATE TO FORMATIVE STUDENT ACHIEVEMENT 
OUTCOMES? 


Student scores on DIBELS Next subtests by grade level are presented in Exhibit 19. In spring 2019, 
between 37% and 57% of students met recommended benchmarks for DIBELS Next subtests, and 
between 0% and 31% of students were at risk. Total time, activity time, number of weeks, average total 
minutes per week, average activity minutes per week, and meeting the optimal dosage threshold were 
the dosage indicators used in the correlational analyses to understand the relationship between dosage 
and student achievement on the DIBELS Next assessments. Dosage indicators were each examined in 
independent regression analyses due to high intercorrelation. All regression findings are presented with 
more detail in the Appendix. 


“The presented optimal activity times are estimates and may not generalize to all /magine Language & Literacy users. 
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EXHIBIT 19. SPRING 2019 SCORES ON THE DIBELS NEXT SUBTESTS BY GRADE LEVEL 


% Meeting 
Recommended DIBELS 
Mean SD Next Benchmark % At Risk 


Letter Naming Fluency (Kindergarten) 57.3. 17.2 37.1 31.3 
Nonsense Word Fluency: Correct Letter Sound (Grade 1) 90.4 38.6 44.3 29.8 
Nonsense Word Fluency: Whole Words Correct (Grade 1) 28.1 15.7 47.5 27.6 
Oral Reading Fluency: Words Correct (Grade 2) 104.2 35.7 42.6 25.5 
Oral Reading Fluency: Accuracy (Grade 2) 96.7 6.4 56.6 0.0 


Increased program dosage was associated with higher student achievement for students in 
Kindergarten and Grade 1 after accounting for prior achievement and student characteristics. All 
indicators of program dosage, including total time, total activity time, total weeks of participation, 
average total minutes per week, average activity minutes per week, and meeting the recommended 
optimal dosage for total minutes per week (Exhibit 20) were significantly and positively related to higher 
spring scores on the subscales for DIBELS Next Letter Naming Fluency (K), and on Nonsense Word 
Fluency Correct Letter Sounds and Whole Words Correct (Grade 1). No indicators for Grade 2 were 
significantly related to spring DIBELS Next scores on the Oral Reading Fluency: Words Correct subscale 
or the Oral Reading Fluency: Accuracy subscale. 


EXHIBIT 20. RELATIONSHIP BETWEEN DOSAGE AND STUDENT ACHIEVEMENT 


DIBELS Next Kindergarten Grade 1 Grade 2 


Letter Naming Fluency Positive 


Nonsense Word Fluency: 


Positive 
Correct Letter Sounds 


Nonsense Word Fluency: 


Whole Words Correct Positive 


Oral Reading Fluency: Words 
Correct 


n.s. 


Oral Reading Fluency: Accuracy ns. 


@n.s. = Not significant at the p < .05 level.’> Shaded boxes indicate the DIBELS Next subtest was not administered to 
that grade. 

Note. For each outcome measure and grade level, the effect of the implementation factor for Imagine Language & 

Literacy program is noted as not significant, positive, or negative. Dosage indicators include total time, total activity 
time, total weeks of participation, average total minutes per week, average activity minutes per week, and meeting 
the recommended optimal dosage threshold for total minutes per week. 


More details of the strength of the relationships for students in Kindergarten are presented in Exhibits 
21 through 23. As seen in Exhibit 21, when converted to hours, for students in Kindergarten, an 
additional hour of participation was associated with just over a third of a point higher score (0.006 x 60 
minutes) on the DIBELS Next Letter Naming Fluency subtest. An additional hour of activity time within 
Imagine Language & Literacy was associated with almost a half-point increase (0.008 x 60 minutes) in 


15 p-value is an indicator that represents the likelihood that observed results occurred by chance. In education research, values 
of p < .05 (i.e., values indicating that observed results had a less than 5% chance of occurring by chance) are typically used to 
identify results that are statistically significant. Lower p-values indicate a smaller likelihood that observed results occurred by 
chance and are therefore associated with statistically significant findings. 


RMC Research Corporation, Denver, CO 30 Study of Imagine Language & Literacy 
in Grades K-2 


score on the DIBELS Next Letter Naming Frequency subtest. An additional week of participation was 
associated with almost a 0.4-point higher score on the DIBELS Next Letter Naming Fluency subtest. An 
additional 5 minutes of total time per week was associated with a 1.4-point higher score (0.287 x 5 
minutes) and 5 minutes additional activity time per week was associated with an almost 1.8-point higher 
score (0.356 x 5 minutes) on the DIBELS Next Letter Naming Fluency subtest. Meeting the recommended 
optimal dosage threshold of 45 minutes per week was associated with 6.5-point higher score on the 
Letter Naming Fluency subtest. 


RMC Research Corporation, Denver, CO 31 Study of Imagine Language & Literacy 
in Grades K-2 


EXHIBIT 21. DOSAGE AND SPRING 2019 DIBELS NeXT LETTER NAMING FLUENCY (LNF) SCORES, KINDERGARTEN 


Model F: Met Total 


Minutes Per Week 
Model A: Total Time Model B: Activity Time Model C: Weeks of Model D: Total 


Model E: Activity Optimal Dosage 
(Minutes) (Minutes) Participation Minutes Per Week Minutes Per Week Threshold 
Estimate 6 SEY Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE 
Intercept 36.560*** 2.014 36.753 *** 2.072 34.863*** 2.953 | 31.095*** 2.612 31.549*** 2.726 40.392*** 1.576 
2018 Fall DIBELS 5B5*** .041 5O2*** .041 .608*** .041 5 74*** .041 583*** 041 .610*** .041 
Next LNF Score 
Dosage .006*** .001 .008*** .002 .371*** .105 .287*** .049 .356*** .067 6.506*** 1.302 


*#*p < 001. 


Note. Separate models were used to examine each dosage indicator. Though the coefficients for total time and total activity time are small, this is to be expected as 
increases are documented in 1-minute increments. 


16 The table contains unstandardized regression coefficients (B) for each of the independent variables. 
1” The standard error (SE) is a measure of the amount of sampling error associated with each regression coefficient. 
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In summary, an increase of 167 total minutes, 125 activity minutes, 2.7 weeks, 3.5 minutes per week, or 
2.8 activity minutes per week was associated with a one-point higher score on the Kindergarten Letter 
Naming Fluency scores (Exhibit 22). For Kindergarten students in the analytic sample, a one-point higher 
score on this measure would be related to an additional 1.8% of students meeting DIBELS Next 
benchmark scores for progress and 2.6% fewer students being identified as at risk. The dosages 
associated with a one-to-five-point higher score on DIBELS Next Letter Naming Fluency can be found in 
Exhibit 23 with the associated relationship with the percent of students meeting DIBELS Next benchmark 
scores and being identified as at risk. 


EXHIBIT 22. SUMMARY OF DOSAGE INDICATORS ASSOCIATED 
WITH ONE-POINT HIGHER DIBELS NExT SCORE, KINDERGARTEN 


167 minutes 


125 activity minutes e@ 
eccooe, @ 


Grade K 
2.7 a ‘ One-point DIBELS Next 
increase Letter Naming 


Fluency 


3.5 minutes/week 
eee0e@ 


2.8 activity minutes/week @ 
eeee0@ 


EXHIBIT 23. DOSAGE ASSOCIATED WITH HIGHER SCORES ON THE 
DIBELS NeXT LETTER NAMING FLUENCY (LNF), KINDERGARTEN 


Additional 
% Meeting Decrease 
Activity Total Activity DIBELS in % 
Changein Total Time Time Number of Minutes Minutes Per Next Identified 
Score (Minutes) (Minutes) Weeks Per Week Week Benchmark as At Risk 
1 167 125 2.7 3.5 2.8 1.8 2.6 
2 333 250 5.4 7.0 5.6 5.1 5.7 
3 500 375 8.1 10.5 8.4 8.5 75 
4 667 500 10.8 13.9 11.2 11.3 8.9 
5 833 625 13.5 17.4 14.0 14.0 9.7 


As seen in Exhibits 24 and 25, for students in Grade 1, an additional hour of participation was associated 
with just under a half-point higher score (0.008 x 60 minutes) on the DIBELS Next Nonsense Word 
Fluency: Correct Letter Sounds subtest and almost a fourth-point higher (0.004 x 60 minutes) on the 
DIBELS Next Nonsense Word Fluency: Whole Words Correct subtest. An additional hour of activity time 
was associated with just over a half point higher (0.009 x 60 minutes) on the DIBELS Next Nonsense 
Word Fluency: Correct Letter Sounds subtest and almost a fourth-point higher (0.004 x 60 minutes) on 
the DIBELS Next Nonsense Word Fluency: Whole Words Correct subtest. An additional week of 
participation was associated with scoring 0.7 points higher on the DIBELS Next Nonsense Word Fluency: 
Correct Letter Sounds subtest and a third-point higher score on the DIBELS Next Nonsense Word 
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Fluency: Whole Words Correct subtest. An additional 5 minutes spent interacting with Imagine 
Language & Literacy per week was associated with 1.3 point higher score (0.269 x 5 minutes) on the 
DIBELS Next Nonsense Word Fluency: Correct Letter Sounds subtest and a 0.7 point higher score (0.136 
x 5 minutes) on the DIBELS Next Nonsense Word Fluency: Whole Words Correct subtest. An additional 

5 activity minutes per week was associated with 1.6-points higher (0.316 x 5 minutes) on the DIBELS 
Next Nonsense Word Fluency: Correct Letter Sounds subtest and over an eighth of a point higher score 
(0.161 x 5 minutes) on the DIBELS Next Nonsense Word Fluency: Whole Words Correct subtest. Meeting 
the recommended optimal dosage threshold of 60 minutes per week was associated with almost a 

5.8 points higher score on the DIBELS Next Nonsense Word Fluency: Correct Letter Sounds subtest and a 
3.3 points higher score on the DIBELS Next Nonsense Word Fluency: Whole Words Correct subtest. 
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EXHIBIT 24. DOSAGE AND SPRING 2019 DIBELS NEXT NONSENSE WORD FLUENCY: 
CORRECT LETTER SOUNDS (NWF: CLS) SCORES, GRADE 1 


Model A: Total Time 


Model B: Activity Time 


Model C: Weeks of 


Model D: Total 


Model E: Activity 


Model F: Met Total 


Minutes Per Week 
Optimal Dosage 


(Minutes) (Minutes) Participation Minutes Per Week Minutes Per Week Threshold 
Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE 
Intercept 47.074*** 4.196 | 47.785*** 4.184 | 40.027*** 6.846  43.427*** 5.020 44.845*** 4.978 55.130*** 3.353 
2018 Fall DIBELS B44*** .043 .842*** .043 852*** .043 .838*** .043 .836*** 0.43 B42*** .043 
Next NWF: CLS 
Score 
Dosage .008*** -002 .009*** -003 692** 241 .269*** .075 .316*** .096 5.774* 2.503 


*p <.05, **p <.01, ***p < .001. 

Note. Separate models were used to examine each dosage indicator. All models account for prior achievement (Fall DIBELS Next), gender, race, special education 
eligibility, and English language learner status. Though the coefficients for total time and total activity time are small, this is to be expected as increases are 
documented in 1-minute increments. 


EXHIBIT 25. DOSAGE AND SPRING 2019 DIBELS NEXT NONSENSE WORD FLUENCY: 
WHOLE WorpDs CorRRECT SOUNDS (NWF: WRC) Scores, GRADE 1 


Model F: Met Total 
Minutes Per Week 


Model A: Total Time Model B: Activity Time | Model C: Weeks of Model D: Total Model E: Activity Optimal Dosage 
(Minutes) (Minutes) Participation Minutes Per Week Minutes Per Week Threshold 
Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE 
Intercept 19.081*** 1.563 19.389*** 1.560 16.306*** 2.746  16.911*** 1.966 17.526*** 1.957 22.628*** 1.156 
2018 Fall DIBELS 31 397** .043 .738*** .043 A5Zts* .043 13 4*** .043 A31*** .044 74 Q*** .044 
Next NWF: WRC 
Score 
Dosage .004*** -001 .004*** -001 311** 102 .136*** .032 .161*** -041 3.316** 1.065 


**o < .01, ***p < .001. 

Note. Separate models were used to examine each dosage indicator. All models account for prior achievement (Fall DIBELS Next), gender, race, special education 
eligibility, and English language learner status. Though the coefficients for total time and total activity time are small, this is to be expected as increases are 
documented in 1-minute increments. 
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In summary, an increase of 125 total minutes, 111 activity minutes, 1.4 weeks, an additional 3.7 minutes 
per week or 3.2 activity minutes per week was associated with a one-point higher score on the Grade 1 
DIBELS Next Nonsense Word Fluency: Correct Letter Sounds subtest (Exhibit 26). For Grade 1 students in 
the analytic sample, a one-point higher score on this measure would be related to an additional half 
percent of students meeting DIBELS Next benchmark scores for progress and 0.2% fewer students being 
identified as at risk. An increase of 250 total minutes, 250 activity minutes, 3.2 weeks, 7.4 total minutes 
per week or 6.2 activity minutes per week was associated with a one-point higher score on the Grade 1 
DIBELS Next Nonsense Word Fluency: Whole Words Correct subtest (Exhibit 26). For Grade 1 students in 
the analytic sample, a one-point higher score on this measure would be related to an additional 1.1% of 
students meeting DIBELS Next benchmark scores for progress and 2.8% fewer students being identified 
as at risk. The dosages associated with a one-to-five-point higher DIBELS Next Nonsense Word Fluency: 
Correct Letter Sounds scores can be found in Exhibit 27 and in Exhibit 28 for the DIBELS Next Nonsense 
Word Fluency: Whole Words Correct scores along with the associated impact on the percent of students 
meeting DIBELS Next benchmark scores and being identified as at risk. 


EXHIBIT 26. SUMMARY OF DOSAGE INDICATORS ASSOCIATED WITH 
ONE-POINT INCREASE IN DIBELS NEXT SCORE, GRADE 1 


125 total minutes 
ee 


111 activity minutes ® 
eee00 


@ Grade 1 


1.4 weeks One- DIBELS Next 
ceooeoe® _ point Nonsense Word Fluency: 
Increase Correct Letter Sounds 
3.7 minutes/week gS 


3.2 activity minutes/week @ 
eee0e@ 


250 total minutes 
e 


250 activity minutes 


Grade 1 
One- DIBELS Next 


3.2 weeks / 
@e0ee0 @ _ point Nonsense Word Fluency: 
UCR ASE Whole Words Correct 
7.4 minutes/week © 
@©ee00600 


6.2 activity minutes/week © 
eeee0 
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EXHIBIT 27. DOSAGE ASSOCIATED WITH HIGHER SCORES ON THE 
DIBELS NEXT NONSENSE WORD FLUENCY: CORRECT LETTER SOUND, GRADE 1 


Total Activity Total Activity Additional Decrease in 
Time Time Number of Minutes per Minutes Per % Meeting % Identified 
(Minutes) (Minutes) Weeks Week Week Threshold as At Risk 
1 125 111 1.4 3.7 3.2 0.5 0.2 
2 250 222 2.9 7.4 6.3 1.3 .09 
3 375 333 4.3 11.2 9.5 2.0 1.7 
4 500 444 5.8 14.9 12.7 2.4 3.5 
5 625 556 7.2 18.6 15.8 2.6 5.2 
EXHIBIT 28. DOSAGE ASSOCIATED WITH HIGHER SCORES ON THE 
DIBELS NEXT NONSENSE WORD FLUENCY: WHOLE WORD CoRRECT, GRADE 1 
Total Activity Total Activity Additional Decrease in 
Time Time Number of Minutes per Minutes Per % Meeting % Identified 
(Minutes) (Minutes) Weeks Week Week Threshold as At Risk 
1 250 250 3.2 7.4 6.2 1.1 2.8 
2 500 500 6.4 14.7 12.4 2.6 4.5 
3 750 750 9.6 22.1 18.6 3.7 6.6 
4 1,000 1,000 12.9 29.4 24.8 5.4 7.9 
5 1,250 1,250 16.1 36.8 31.1 6.7 9.6 


For students in Grade 2, total time, total activity time, number of weeks of participation, average 
minutes per week, average activity minutes per week, and meeting the recommended optimal threshold 
for dosage in Imagine Language & Literacy were not significantly associated with spring DIBELS Next 
scores on the DIBELS Next Oral Reading Fluency: Whole Words Correct subtest after accounting for prior 
achievement (Fall DIBELS Next), gender, race, special education eligibility, and English language learner 
status (Exhibit 29). Similarly, for these students, total time, total activity time, number of weeks of 
participation, average minutes per week, average activity minutes per week, and meeting the 
recommended optimal threshold for dosage in Imagine Language & Literacy were not significantly 
associated with DIBELS Next scores on the DIBELS Next Oral Reading Fluency: Accuracy subtest after 
accounting for prior achievement (Fall DIBELS Next), gender, race, special education eligibility, and 
English language learner status (Exhibit 30). 
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EXHIBIT 29. DOSAGE AND SPRING 2019 DIBELS NEXT ORAL READING FLUENCY: 
Worps Correct (ORF: WC) Scores, GRADE 2 


Model A: Total Time 


Model B: Activity Time 


Model C: Weeks of 


Model D: Total 


Model E: Activity 


Model F: Met Total 
Minutes Per Week 
Optimal Dosage 


(Minutes) (Minutes) Participation Minutes Per Week Minutes Per Week Threshold 
Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE 

Intercept 40.797*** 3.046 | 40.116*** 3.018 | 37.615*** 3.858 41.817*** 3.548 40.568*** 3.490 41.751*** 2.497 
2018 Fall DIBELS .912*** 025 .913*** 025 .912*** 025 .911*** 025 .912*** 025 .908*** 025 
Next ORF: WC 

Score 

Dosage -000 .001 .001 .002 .128 118 -.016 .047 .009 .061 -2.041 1.860 

**#y < 001. 


Note. Separate models were used to examine each dosage indicator. All models account for prior achievement (Fall DIBELS Next), gender, race, special education 
eligibility, and English language learner status. Though the coefficients for total time and total activity time are small, this is to be expected as increases are 


documented in 1-minute increments. 


EXHIBIT 30. DOSAGE AND SPRING 2019 DIBELS NEXT ORAL READING FLUENCY: 
Accuracy (ORF: ACC) Scores, GRADE 2 


Model F: Met Total 
Minutes Per Week 


Model A: Total Time Model B: Activity Time | Model C: Weeks of Model D: Total Model E: Activity Optimal Dosage 
(Minutes) (Minutes) Participation Minutes Per Week Minutes Per Week Threshold 
Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE 
Intercept 56.540*** 1.581 | 56.464*** 1.572 | 56.093*** 1.629 | 56.839*** 1.631 56.617*** 1.615 56.952*** 1.555 
2018 Fall DIBELS 435 *** .016 434*** .016 434*** .016 435*** .016 435 *** .016 434*** .016 
Next ORF: ACC 
Score 
Dosage .000 .000 .000 .000 .031 .025 -.002 .010 .004 .013 -.391 -402 
*** < 001. 


Note. Separate models were used to examine each dosage indicator. All models account for prior achievement (Fall DIBELS Next), gender, race, special education eligibility, 
and English language learner status. Though the coefficients for total time and total activity time are small, this is to be expected as increases are documented in 1-minute 


increments. 
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CONCLUSIONS AND RECOMMENDATIONS 


RMC Research used a case study approach to document implementation of Imagine Language & 
Literacy and a correlational design to examine the relationship between student program usage and 
student academic achievement outcomes for students in Grades K-2. The study was conducted during 
the 2018/19 academic year at eight schools in two school districts in the Southeastern United States. 
The case study approach involved interviews with administrators, focus groups with teachers, and 
observations of Imagine Language & Literacy in six schools to gather detailed information about 
implementation. For all study schools, RMC Research and Imagine Learning collected usage data and 
administrative data including demographic information and scores on the formative DIBELS Next 
assessment for all students in Grades K-2 to examine the relationship between student program usage 
and student academic achievement. Findings focus on implementation factors and the impact of the 
amount of instruction received in Imagine Language & Literacy on spring DIBELS Next scores. 


FINDINGS 


In general, teachers and administrators found the professional development provided to be sufficient 
for beginning implementation of Imagine Language & Literacy. Teachers and administrators reported 
that the initial professional development they received from Imagine Learning was informative and 
sufficient to get them started using Imagine Language & Literacy. 


Teachers suggested additional training or documentation on using Imagine Language & Literacy. 
Teachers indicated they would have liked additional training on how to use many of the components of 
Imagine Language & Literacy and that they would specifically like training regarding the student-level 
reports available in the system. Additionally, some teachers requested a “Frequently Asked Questions 
(FAQ)” or other documentation be available to aid them in finding information within Imagine Language 
& Literacy such as reports and additional resources. 


Implementation was hindered by technology problems. All teachers and administrators reported 
experiencing technology problems, many of which negatively affected implementation of Imagine 
Language & Literacy. 


Students enjoyed engaging with Imagine Language & Literacy. Teachers and administrators reported 
their students enjoyed working in Imagine Language & Literacy and the program was engaging for 
students. 


Fewer than 60% of students in each grade level met the recommendation for average number of 
minutes per week of participation. Imagine Learning recommends that students in Kindergarten engage 
with Imagine Language & Literacy for at least 45 minutes per week and students in Grades 1 and 2 
engage with the program for at least 60 minutes per week. On average, 56% of Kindergarten students, 
47% of students in Grade 1, and 24% of students in Grade 2 met the recommended total minutes per 
week. 
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Average student participation varied by school. Across the schools in the study, participation in Imagine 
Language & Literacy varied. School averages for total weeks ranged from 15 to 30 weeks, averages for 
total minutes ranged from 647 to just under 2,000 minutes, and averages for activity minutes ranged 
from 507 to 1,604 minutes. 


Extent of participation (dosage) was related to performance on the spring DIBELS Next assessment for 
students in Kindergarten. All indicators of dosage were positively related to spring scores on the DIBELS 
Next assessment for students in Kindergarten. For example, after accounting for prior achievement and 
demographic characteristics, an increase in participation of 167 total minutes, 125 activity minutes, 

2.7 weeks, 3.5 minutes per week, or 2.8 activity minutes per week was associated with a one-point 
increase on the DIBELS Next Letter Naming Fluency test for students in Kindergarten. For students in the 
analytic sample, a one-point increase on this test would be related to an additional 1.8% of students 
meeting DIBELS Next benchmark scores for progress and 2.6% fewer students being identified as at risk. 


Extent of participation (dosage) was related to performance on the spring DIBELS Next assessment for 
students Grade 1. All indicators of dosage were positively related to spring scores on the DIBELS Next 
assessment for students in Grade 1. For students in Grade 1, an increase of 125 total minutes, 111 
activity minutes, 1.4 weeks, 3.7 minutes per week, or 3.2 activity minutes per week was associated with 
a one-point increase in DIBELS Next Nonsense Word Fluency: Correct Letter Sounds score and increases 
in participation of 250 total minutes, 250 activity minutes, 3.2 weeks, 7.4 minutes per week, or 6.2 
activity minutes per week were associated with a similar increase on the DIBELS Next Nonsense Word 
Fluency: Whole Words Correct score. For students in the analytic sample, a one-point increase on this 
test would be related to an additional half percent of students meeting DIBELS Next benchmark scores 
for progress and 0.2% fewer students being identified as at risk on the DIBELS Next Nonsense Word 
Fluency: Correct Letter Sounds test and an additional 1.1% of students meeting DIBELS Next benchmark 
scores for progress and 2.8% fewer students being identified as at risk on the DIBELS Next Nonsense 
Word Fluency: Whole Word Correct test. 


Extent of participation (dosage) was not significantly related to performance on the spring DIBELS 
Next assessment for students in Grade 2. No indicators of dosage were found to be significantly related 
to spring scores on the DIBELS Next assessment for students in Grade 2. Overall implementation was 
lowest in Grade 2, with only 24% of students meeting optimal dosage recommendations. This finding 
suggests that weak implementation reduces the likelihood of detecting student outcomes. 


RECOMMENDATIONS 


Based on findings from this study, RMC Research offers the following recommendations to guide 
program development and improvement, and to assess the impact of Imagine Language & Literacy. 


1. Provide additional hands-on experience with Imagine Language & Literacy programming and 
reports during the Getting Started training. Recommendations from teachers and administrators 
included providing additional training on how to use each of the components of Imagine Language 
& Literacy, including student reports. 


2. Document technology issues and share common solutions or a manual for common issues. 
Technology issues were a challenge faced by all sites. Consider collecting more systematic 
information about technological issues, particularly among newly implementing sites. Such 
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information may be used to identify the most prevalent issues and types of technical support that 
may be provided to facilitate successful implementation. For example, to support common 
challenges such as incompatibility of microphones or challenges with students logging in to the 
program, a manual or “Frequently Asked Questions (FAQ)” document might be developed so 
teachers and administrators can independently troubleshoot issues and find solutions. 


3. Continue monitoring implementation throughout the academic year. Given variation in 
implementation across grade levels and schools, Imagine Learning may wish to further explore ways 
to regularly monitor usage statistics through the school year. For example, weekly reports of usage 
statistics across and within schools might be reviewed by Imagine Learning and used as a tool for 
communication with sites to assess factors related to low implementation. Imagine Learning may 
also wish to encourage school administrators to monitor implementation and take steps to address 
issues when target dosage thresholds are not met. 


4. Continue encouraging sites to implement programming and meet usage recommendations. 
Students who met optimal dosage for time on program and those with greater overall program 
participation had higher spring achievement scores. Overall implementation was lowest in Grade 2 
and no significant associations with outcomes were found among Grade 2 students, suggesting—not 
surprisingly—that weak implementation reduces the likelihood of detecting student outcomes. 
Encouraging sites to ensure that optimal dosage thresholds are met for all students may help 
improve outcomes. Examining implementation and outcomes for subgroups of students also 
appears to be important. 


LIMITATIONS 


This study includes findings based on case study and correlational evidence. While these findings 
suggest positive relationships between implementation of Imagine Language & Literacy and student 
achievement, no causal conclusions can be drawn. Because different subtests are used to calculate the 
DIBELS Next composite score depending on the grade level and time of year, composite scores are not 
directly comparable across administrations and grade levels (Dynamic Measurement Group, Inc., 2010). 
Therefore, all analyses in this report include only those subtests that were taken in both fall and spring 
in each grade level and do not reflect all content domains measured by DIBELS Next. In addition, 
approximately 1.7% of Imagine Language & Literacy users were excluded from these analyses due to 
low usage (less than 1 month); another approximately 20% were excluded due to missing demographic 
or DIBELS Next data or because usage data could not be matched to demographic or achievement data. 
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APPENDIX 


This Appendix presents tables with full results from the regression analyses, including all demographic variables. 
EXHIBIT A1. DOSAGE AND SPRING 2019 DIBELS NeXT LETTER NAMING FLUENCY (LNF) SCORES, KINDERGARTEN (N = 450) 


Model F: Met Total 
Minutes Per Week 


Model A: Total Time Model B: Activity Time | Model C: Weeks of Model D: Total Model E: Activity Optimal Dosage 
(Minutes) (Minutes) Participation Minutes Per Week Minutes Per Week Threshold 
Estimate ?® SE19 Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE 
Intercept 36.560*** 2.014 36.753 *** 2.072 34.863*** 2.953 31.095*** 2.612 31.549*** 2.726 40.392*** 1.576 
2018 Fall DIBELS 585*** .041 592*** .041 .608*** 041 5 74*** .041 583 *** .041 .610*** .041 
Next LNF Score 
Dosage .006*** .001 .008*** .002 .371*** 105 .287*** .049 .356*** .067 6.506*** 1.302 
Female 271 1.265 301 1.271 .007 1.284 441 1.256 559 1.266 -.074 1.266 
Minority 926 1.342 1.098 1.366 .666 1.387 .686 1.312 1.032 1.342 .272 1.317 
Special Education -10.145*** 2.306 | -10.120*** 2.319 -10.500*** 2.344 | -10.420*** 2.280 | -10.321*** 2.297 | -11.075*** 2.295 
English Language -6.843*** 2.110 -6.989*** 2.136 -6.480** 2.173 -5.740** 2.049 -6.054** 2.073 -5.865** 2.078 
Learner 
R220 .416 411 398 425 418 414 


**n < 01, ***p < 001. 
Note. Separate models were used to examine each dosage indicator. Though the coefficients for total activity time are small, this is to be expected as increases are 
documented in 1-minute increments. 


18 The table contains unstandardized regression coefficients (B) for each of the independent variables. 

19 The standard error (SE) is a measure of the amount of sampling error associated with each regression coefficient. 

20 R? is a measure of the proportion of variance of a dependent measure that can be explained by one or more independent variables in a regression model. Values for this 
statistic range from 0 to 1. 
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EXHIBIT A2. DOSAGE AND SPRING 2019 DIBELS NEXT NONSENSE WORD FLUENCY: 
CORRECT LETTER SOUNDS (NWF: CLS) ScoRES, GRADE 1 (N = 467) 


Model F: Met Total 
Minutes Per Week 


Model A: Total Time Model B: Activity Time Model C: Weeks of Model D: Total Model E: Activity Optimal Dosage 
(Minutes) (Minutes) Participation Minutes Per Week Minutes Per Week Threshold 
Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE 
Intercept 47.074*** 4.196 | 47.785*** 4.184 40.027*** 6.846 | 43.427*** 5.020 44.845*** 4.978 55.130*** 3.353 
2018 Fall DIBELS B44*** .043 .842*** .043 .852** 043 .838*** .043 .836*** .043 842*** .043 
Next NWF: CLS 
Score 
Dosage .008* ** .002 .009*** .003 .692*** 241 .269*** .075 .316*** .096 5.774* 2.503 
Female 1.395 2.518 1.338 2.524 1.365 2.537 1.569 2.520 1.481 2.527 1.713 2.540 
Minority -7.307** 2.541 -7.007** 2.562 -6.931** 2.598 -8.072*** 2.526 -7.618** 2.545 -8.090** 2.552 
Special Education -21.027*** 4.070 | -20.820*** 4.074 | -20.413*** 4.088 21.034*** 4.078 | -20.787*** 4.083 -19.832*** 4.092 
English Language -12.153** 3.973 -12.196** 3.989  -11.783** 4.010 | -11.409** 3.953 -11.459** 3.970 -10.832** 3.987 
Learner 
R? 540 538 534 539 537 531 


*p < .05, **p < .01, ***p < .001. 
Note. Separate models were used to examine each dosage indicator. Though the coefficients for total activity time are small, this is to be expected as increases are documented 
in 1-minute increments. 
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EXHIBIT A3. DOSAGE AND SPRING 2019 DIBELS NEXT NONSENSE WORD FLUENCY: 
WHOLE WorbDs CorRRECT SOUNDS (NWF: WRC) Scores, GRADE 1 (N=467) 


Model F: Met Total 
Minutes Per Week 


Model A: Total Time Model B: Activity Time Model C: Weeks of Model D: Total Model E: Activity Optimal Dosage 
(Minutes) (Minutes) Participation Minutes Per Week Minutes Per Week Threshold 
Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE 
Intercept 19.081*** 1.563 | 19.389*** 1.560 | 16.306*** 2.746 16.911*** 1.966 17.526*** 1.957 22.628*** 1.156 
2018 Fall DIBELS 739% ** .043 .738*** 043 753 *** 043 13 4*** 043 AQ1*** 044 744*** 044 
Next NWF: WRC 
Score 
Dosage .004*** .001 .004*** .001 .311** .102 .136*** .032 .161*** .041 3.316** 1.065 
Female 1.060 1.069 1.031 1.072 1.079 1.081 1.137 1.068 1.090 1.072 1.216 1.077 
Minority -3.536*** 1.076 | -3.388** 1.085 | -3.403** 1.103 -3.893*** 1.068 -3.658*** 1.077 -3.863*** 1.080 
Special Education -10.238*** 1.718 | -10.139*** 1.721  -9.890*** 1.731 | -10.258*** 1.719 | -10.134*** 1.722 -9.623*** 1.724 
English Language -6.394*** 1.669 -6.414*** 1.677  -6.146*** 1.692 -6.051*** 1.657 -6.077*** 1.666 -5.814*** 1.672 
Learner 
R? 499 497 .490 499 496 -490 


*#0 < 01, ***p < .001. 
Note. Separate models were used to examine each dosage indicator. Though the coefficients for total activity time are small, this is to be expected as increases are 
documented in 1-minute increments. 
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EXHIBIT A4. DOSAGE AND SPRING 2019 DIBELS NEXT ORAL READING FLUENCY: 
Worbps Correct (ORF: WC) Scores, GRADE 2 (N = 458) 


Model F: Met Total 
Minutes Per Week 


Model A: Total Time Model B: Activity Time Model C: Weeks of Model D: Total Model E: Activity Optimal Dosage 
(Minutes) (Minutes) Participation Minutes Per Week Minutes Per Week Threshold 
Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE Estimate SE 
Intercept 40.797*** 3.046 | 40.116*** 3.018 37.615*** 3.858 | 41.817*** 3.548 40.568*** 3.490 | 41.751*** 2.497 
2018 Fall DIBELS .912*** .025 .913*** .025 .912*** .025 .911*** .025 912*** .025 .908*** .025 
Next ORF: WC 
Score 
Dosage .000 .001 .001 .002 .128 118 -.016 .047 .009 .061 -2.041 1.860 
Female .784 1.565 .833 1.566 873 1.562 .739 1.565 794 1.567 .769 1.559 
Minority -6.427*** 1.677 -6.215*** 1.684 -5.686*** 1.736 -6.573*** 1.616 -6.416*** 1.622 -6.785*** 1.609 
Special Education -16.700*** 2.631 | -16.700*** 2.630 -16.696*** 2.627 | -16.670*** 2.632 | -16.704*** 2.631 | -16.462*** 2.636 
English Language 4.213 2.690 4.069 2.695 3.819 2.692 4.308 2.677 4.198 2.682 4.529 2.678 
Learner 
R2 .792 .792 .792 .792 .792 .792 


*p <.05, **p <.01, ***p < .001. 
Note. Separate models were used to examine each dosage indicator. Though the coefficients for total activity time are small, this is to be expected as increases are 
documented in 1-minute increments. 
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EXHIBIT A5. DOSAGE AND SPRING 2019 DIBELS NEXT ORAL READING FLUENCY: 
ACCURACY (ORF: ACC) Scores, GRADE 2 (N=445) 


Model F: Met Total 
Minutes Per Week 


Model A: Total Time Model B: Activity Time Model C: Weeks of Model D: Total Model E: Activity Optimal Dosage 
(Minutes) (Minutes) Participation Minutes Per Week Minutes Per Week Threshold 
Estimate SE Estimate SE Estimate SE. Estimate SE Estimate SE Estimate SE 
Intercept 56.540*** 1.581 | 56.464*** 1.572 | 56.093*** 1.629 56.839*** 1.631 56.617*** 1.615 56.952*** 1.555 
2018 Fall DIBELS 435*** .016 434*** .016 434*** .016 435*** .016 435 *** .016 434*** .016 
Next ORF: ACC 
Score 
Dosage .000 .000 .000 .000 .031 .025 -.002 .010 .004 .013 -.391 402 
Female 305 330 317 330 313 330 .284 330 .297 331 .285 329 
Minority -.217 349 -.178 350 -.103 362 -.294 337 -2.66 338 -.333 335 
Special Education -2.748*** .601 -2.756*** .601 -2.760*** 600 | -2.725*** .602 -2.740*** .601 -2.677*** .602 
English Language .968 576 938 577 914 577 1.023 574 998 575 1.069 574 
Learner 
R2 721 721 721 .720 721 721 


*p <.05, **p <.01, ***p < .001. 
Note. Separate models were used to examine each dosage indicator. Though the coefficients for total activity time are small, this is to be expected as increases are 
documented in 1-minute increments. 
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